feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it #3530

sirasistant · 2023-12-04T09:37:38Z

This PR prepares the standard indexed tree to be used for the public data tree. Changes:

Make StandardIndexedTree abstract in leaves in preparation to accomodate other trees than the nullifier tree
Removed the leaves in memory in the StandardIndexedTree and switch to db queries for low leaf finding (O(1))
FindLeafIndex is now O(1) too

Checklist:

Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.

If the pull request requires a cryptography review (e.g. cryptographic algorithm implementations) I have added the 'crypto' tag.
I have reviewed my diff in github, line by line and removed unexpected formatting changes, testing logs, or commented-out code.
Every change is related to the PR description.
I have linked this pull request to relevant issues (if any exist).

AztecBot · 2023-12-04T10:00:56Z

Benchmark results

Metrics with a significant change:

node_database_size_in_bytes (5): 3,628,812 (+30%)
node_database_size_in_bytes (10): 3,914,928 (-22%)
note_history_trial_decrypting_time_in_ms (10): 367 (+59%)
note_trial_decrypting_time_in_ms (8): 17.1 (-36%)
note_trial_decrypting_time_in_ms (32): 39.8 (-36%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 9f682cb8 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,972	867,920	3,449,540
l1_rollup_execution_gas	842,059	3,595,028	22,204,909
l2_block_processing_time_in_ms	2,207	8,455 (-2%)	34,946 (-1%)
note_successful_decrypting_time_in_ms	309 (-2%)	913 (-2%)	3,311 (-5%)
note_trial_decrypting_time_in_ms	⚠️ 17.1 (-36%)	⚠️ 39.8 (-36%)	200 (-1%)
l2_block_building_time_in_ms	20,146 (-1%)	80,155 (-2%)	320,349 (-2%)
l2_block_rollup_simulation_time_in_ms	16,624 (-1%)	66,297 (-2%)	264,734 (-2%)
l2_block_public_tx_process_time_in_ms	3,490	13,788 (-1%)	55,373 (-1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	24,614 (-2%)	48,042 (-1%)
note_history_successful_decrypting_time_in_ms	2,231 (-2%)	4,355 (-1%)
note_history_trial_decrypting_time_in_ms	165 (-1%)	⚠️ 367 (+59%)
node_database_size_in_bytes	⚠️ 3,628,812 (+30%)	⚠️ 3,914,928 (-22%)
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	199	43,109	20,441
private-kernel-ordering	115	25,833	9,689
base-rollup	2,956	667,692 (+1%)	873
root-rollup	84.5 (-8%)	4,072	881
private-kernel-inner	260	64,516	20,441
public-kernel-private-input	172	25,203	20,441
public-kernel-non-first-iteration	168 (-1%)	25,245	20,441
merge-rollup	10.8 (-2%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	10,323	25,938

LeilaWang · 2023-12-04T19:49:44Z

yarn-project/merkle-tree/src/new_tree.ts

+ * @param clazz - The class to be instantiated.
+ * @returns A builder function.
+ */
+export function builder<T>(


Can we call it treeBuilder so that we don't have to change its name when exporting?

LeilaWang · 2023-12-04T19:51:59Z

yarn-project/merkle-tree/src/new_tree.ts

+ * @returns A builder function.
+ */
+export function builder<T>(
+  clazz: new (db: LevelUp, hasher: Hasher, name: string, depth: number, size: bigint, root?: Buffer) => T,


This is very cute 😂

LeilaWang · 2023-12-04T20:03:00Z

yarn-project/merkle-tree/src/snapshots/indexed_tree_snapshot.ts

+
+  async findLeafIndex(value: Buffer): Promise<bigint | undefined> {
+    const index = await this.tree.findLeafIndex(value, false);
+    if (index && index < this.getNumLeaves()) {


Should be index !== undefined?

LeilaWang · 2023-12-04T20:12:01Z

yarn-project/merkle-tree/src/standard_indexed_tree/standard_indexed_tree.ts

+      const cachedLowLeafPreimage =
+        cachedLowLeafIndex !== undefined ? this.getCachedPreimage(cachedLowLeafIndex) : undefined;
+      if (cachedLowLeafIndex && (!lowLeafIndex || cachedLowLeafPreimage!.getKey() > lowLeafPreimage!.getKey())) {


Suggested change

const cachedLowLeafPreimage =

cachedLowLeafIndex !== undefined ? this.getCachedPreimage(cachedLowLeafIndex) : undefined;

if (cachedLowLeafIndex && (!lowLeafIndex || cachedLowLeafPreimage!.getKey() > lowLeafPreimage!.getKey())) {

if (cachedLowLeafIndex !== undefined) {

const cachedLowLeafPreimage = this.getCachedPreimage(cachedLowLeafIndex)!;

if (!lowLeafPreimage || cachedLowLeafPreimage.getKey() > lowLeafPreimage.getKey()) {}

}

LeilaWang · 2023-12-04T20:41:09Z

yarn-project/merkle-tree/src/standard_indexed_tree/standard_indexed_tree.ts

+   * @param treeHeight - Height of tree for sibling path.
+   * @returns An empty witness.
+   */
+  getEmptyLowLeafWitness<N extends number>(treeHeight: N): LowLeafWitnessData<N> {


Shouldn't treeHeight always be this.depth?

Yeah, I included this as a class method but I was facing some generics funkyness when trying to use this.getHeight() as the generic of LowLeafWitnessdata. I'll see if I can get it resolved :D

LeilaWang · 2023-12-04T20:57:24Z

yarn-project/merkle-tree/src/tree_base.ts

+   * @param includeUncommitted - Indicates whether to include uncommitted data.
+   * @returns The index of the first leaf found with a given value (undefined if not found).
+   */
+  public async findLeafIndex(value: Buffer, includeUncommitted: boolean): Promise<bigint | undefined> {


This won't work for sparse tree though, and sparse tree extends this.

Oh, this was previously in the merkle_trees db classes that potentially applied this algorithm for any tree. Let me check...

LeilaWang · 2023-12-04T21:05:54Z

yarn-project/merkle-tree/src/new_tree.ts

  db: LevelUp,
  hasher: Hasher,
  name: string,
  depth: number,
  prefilledSize = 1,
 ): Promise<T> {
-  const tree = new c(db, hasher, name, depth, 0n, undefined);
+  const tree = c(db, hasher, name, depth, 0n);


Can it always use treeBuilder here so we can still pass treeClass to it?

So the issue here was:

newTree and loadTree were called with constructor (typed with new) functions, which all had the same shape before

now indexed trees have different constructors, because they require the leaf factory and the preimage factory

Typescript won't let me use the new operator if the function is not typed as a constructor (which makes sense)

So now newTree and loadTree sometimes need to issue "new" and sometimes not

My solution was ok, let's accept regular functions, and if you are passing a regular constructor just wrap it in a regular function via treeBuilder and you're set. If you are passing a more complex constructor then just wrap it in a closure that provides the factories needed for that specific tree instance.

I tried to type newTree and loadTree with a union type of constructor and regular function, but I didn't find a clean way to narrow the type so it allowed me to do new sometimes and just calling other times

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.2</summary> ## [0.16.2](aztec-packages-v0.16.1...aztec-packages-v0.16.2) (2023-12-05) ### Features * Add tree snapshots ([#3468](#3468)) ([7a86bb3](7a86bb3)) * **AVM:** First version for mini AVM (ADD, RETURN, CALLDATACOPY) ([#3439](#3439)) ([b3af146](b3af146)) * Circuit optimized indexed tree batch insertion ([#3367](#3367)) ([187d2f7](187d2f7)) * Devnet ([#3473](#3473)) ([97c40c2](97c40c2)) * **docs:** Add simple private voting tutorial ([#3402](#3402)) ([a6e0352](a6e0352)) * **docs:** Document slow update tree ([#3416](#3416)) ([8e9f103](8e9f103)) * Flavor refactor, reduce duplication ([#3407](#3407)) ([8d6b013](8d6b013)) * Inclusion and non-inclusion proofs experiment ([#3255](#3255)) ([b911e65](b911e65)), closes [#2572](#2572) [#2584](#2584) * New Poseidon2 circuit builder gates ([#3346](#3346)) ([91cb369](91cb369)) * New Poseidon2 relations ([#3406](#3406)) ([14b9736](14b9736)) * Pull latest noir for brillig optimizations ([#3464](#3464)) ([d356bac](d356bac)) * Refactor StandardIndexedTree for abstract leaves and preimages and optimized it ([#3530](#3530)) ([63b9cdc](63b9cdc)) * Removing historical roots from circuits ([#3544](#3544)) ([9f682cb](9f682cb)) * Seperate pil files for sub machines ([#3454](#3454)) ([d09d6f5](d09d6f5)) * Throw compile time error if contract has too many fns ([#3536](#3536)) ([ad66ad0](ad66ad0)) * Use tree snapshots in aztec-node/pxe/oracles ([#3504](#3504)) ([6e40427](6e40427)) * Yellow paper cross-chain communication ([#3477](#3477)) ([d51df8c](d51df8c)) ### Bug Fixes * Check version, chainid and sender for cross-chain l1 to l2 msgs ([#3457](#3457)) ([d251703](d251703)) * **ci:** Add DEPLOY_TAG in fork log group ([#3510](#3510)) ([f021041](f021041)) * **ci:** Check if l1 contracts img has been deployed ([#3531](#3531)) ([ac1f03c](ac1f03c)) * **ci:** Comment out LB listeners (for now) ([#3519](#3519)) ([640aabc](640aabc)) * **ci:** Count for bootnode discovery service ([#3517](#3517)) ([2a38788](2a38788)) * **ci:** Define REPOSITORY in deploy_l1_contracts ([#3514](#3514)) ([b246d1b](b246d1b)) * **ci:** Don't deploy to npm on master merge ([#3502](#3502)) ([a138860](a138860)) * **ci:** Env vars for deploying l1-contracts ([#3513](#3513)) ([27106b2](27106b2)) * **ci:** Export FORK_API_KEY from setup_env ([#3512](#3512)) ([7e81e2c](7e81e2c)) * **ci:** Fix docker architecture for devnet packages ([#3505](#3505)) ([66d0287](66d0287)) * **ci:** Fix faucet vars + don't deploy contracts from node ([#3553](#3553)) ([c7176f6](c7176f6)) * **ci:** L1 contracts directories ([#3545](#3545)) ([63dd0c8](63dd0c8)) * **ci:** Login to ecr to fetch contracts image ([#3538](#3538)) ([b033538](b033538)) * **ci:** Remove unused ADDRESS vars & export private key vars ([#3520](#3520)) ([d889359](d889359)) * **ci:** Set default value for $TO_TAINT ([#3508](#3508)) ([8b6688a](8b6688a)) * **ci:** Terraform listener resources ([#3534](#3534)) ([c3b9cce](c3b9cce)) * **ci:** Terraform_deploy for devnet ([#3516](#3516)) ([ba3803e](ba3803e)) * **ci:** Tf variable references & formatting([#3522](#3522)) ([d37cf52](d37cf52)) * Disable e2e-slow-tree ([#3459](#3459)) ([5927103](5927103)) * **docs:** Update package name of aztec-cli ([#3474](#3474)) ([98d7ba0](98d7ba0)) * Double slash in deployed faucet routes ([#3555](#3555)) ([6c704a5](6c704a5)) * Faucet lb_listener priority ([#3554](#3554)) ([3f56dd7](3f56dd7)) * Handling low_nullifier.next_value equal to 0 ([#3562](#3562)) ([c800502](c800502)), closes [#3550](#3550) * Remove x86_64 form l1-contracts img tag ([#3549](#3549)) ([6828f1a](6828f1a)) * Throw error if fn sig has whitespaces ([#3509](#3509)) ([7671063](7671063)), closes [#3055](#3055) ### Miscellaneous * (yellow paper) public-vm section of yellow paper ([#3493](#3493)) ([8ff3780](8ff3780)) * Add mermaid diagram support ([#3499](#3499)) ([537d552](537d552)) * Add yellow paper build check to CI ([#3490](#3490)) ([3ebd2f2](3ebd2f2)) * **avm:** Enable AVM unit tests in CI ([#3463](#3463)) ([051dda9](051dda9)), closes [#3461](#3461) * **bb:** Pointer_view to reference-based get_all ([#3495](#3495)) ([50d7327](50d7327)) * **bb:** Reuse entities from GoblinUltra in GoblinUltraRecursive ([#3521](#3521)) ([8259636](8259636)) * Build the acir test vectors as part of CI. ([#3447](#3447)) ([1a2d1f8](1a2d1f8)) * Containers reduced to ~100MB total. ~30s installation. ([#3487](#3487)) ([b49cef2](b49cef2)) * **docs:** Fix broken Noir stdlib link ([#3496](#3496)) ([787d59a](787d59a)) * Field-agnostic and reusable transcript ([#3433](#3433)) ([d78775a](d78775a)) * Fix broken link in txs in yellow paper ([#3484](#3484)) ([798565d](798565d)) * Fix yellow paper build error ([32881a4](32881a4)) * Fixed typo in build system ([#3501](#3501)) ([3a80ac2](3a80ac2)) * Increase functions per contract from 16 to 32 ([#3503](#3503)) ([ebdeea3](ebdeea3)) * Naming fixes ([#3476](#3476)) ([1db30bf](1db30bf)) * Optimise bb.js package size and sandox/cli dockerfiles to unbloat final containers. ([#3462](#3462)) ([cb3db5d](cb3db5d)) * Pin node version in docker base images and bump nvmrc ([#3537](#3537)) ([5d3895a](5d3895a)) * Recursive verifier updates ([#3452](#3452)) ([dbb4a12](dbb4a12)) * Refactor `WitnessEntities` to be able to derive `WitnessCommitments` from it ([#3479](#3479)) ([9c9b561](9c9b561)) * Remove temporary logging ([#3466](#3466)) ([8c8387b](8c8387b)) * Transcript handled through shared_ptr ([#3434](#3434)) ([30fca33](30fca33)) * Typo fixes ([#3488](#3488)) ([d9a44dc](d9a44dc)) * **yellow_paper:** Public<>private messaging ([#3491](#3491)) ([6ecc406](6ecc406)) ### Documentation * Add transaction section to yellow paper ([#3418](#3418)) ([44bf30b](44bf30b)) * Apply comments from Jan on contracts ([#3539](#3539)) ([e351873](e351873)) * Fees update in yellow paper ([#3486](#3486)) ([a8b2608](a8b2608)) * First go at generated AVM instruction set doc ([#3469](#3469)) ([8cc54a4](8cc54a4)) * Further update to the yellow paper ([#3542](#3542)) ([751bb6a](751bb6a)) * Yellow paper updates ([#3478](#3478)) ([11f754d](11f754d)) * Yellow paper updates for private message delivery ([#3472](#3472)) ([6ba9e18](6ba9e18)) * **yellow-paper:** Sync, enqueued, and static calls ([#3494](#3494)) ([00835c6](00835c6)), closes [#3108](#3108) * **yellowpaper:** Instruction set updates and fixes ([#3515](#3515)) ([bfb61dd](bfb61dd)) </details> <details><summary>barretenberg.js: 0.16.2</summary> ## [0.16.2](barretenberg.js-v0.16.1...barretenberg.js-v0.16.2) (2023-12-05) ### Miscellaneous * Optimise bb.js package size and sandox/cli dockerfiles to unbloat final containers. ([#3462](#3462)) ([cb3db5d](cb3db5d)) * Pin node version in docker base images and bump nvmrc ([#3537](#3537)) ([5d3895a](5d3895a)) </details> <details><summary>barretenberg: 0.16.2</summary> ## [0.16.2](barretenberg-v0.16.1...barretenberg-v0.16.2) (2023-12-05) ### Features * **AVM:** First version for mini AVM (ADD, RETURN, CALLDATACOPY) ([#3439](#3439)) ([b3af146](b3af146)) * Flavor refactor, reduce duplication ([#3407](#3407)) ([8d6b013](8d6b013)) * New Poseidon2 circuit builder gates ([#3346](#3346)) ([91cb369](91cb369)) * New Poseidon2 relations ([#3406](#3406)) ([14b9736](14b9736)) * Pull latest noir for brillig optimizations ([#3464](#3464)) ([d356bac](d356bac)) * Seperate pil files for sub machines ([#3454](#3454)) ([d09d6f5](d09d6f5)) ### Miscellaneous * **avm:** Enable AVM unit tests in CI ([#3463](#3463)) ([051dda9](051dda9)), closes [#3461](#3461) * **bb:** Pointer_view to reference-based get_all ([#3495](#3495)) ([50d7327](50d7327)) * **bb:** Reuse entities from GoblinUltra in GoblinUltraRecursive ([#3521](#3521)) ([8259636](8259636)) * Build the acir test vectors as part of CI. ([#3447](#3447)) ([1a2d1f8](1a2d1f8)) * Field-agnostic and reusable transcript ([#3433](#3433)) ([d78775a](d78775a)) * Optimise bb.js package size and sandox/cli dockerfiles to unbloat final containers. ([#3462](#3462)) ([cb3db5d](cb3db5d)) * Pin node version in docker base images and bump nvmrc ([#3537](#3537)) ([5d3895a](5d3895a)) * Recursive verifier updates ([#3452](#3452)) ([dbb4a12](dbb4a12)) * Refactor `WitnessEntities` to be able to derive `WitnessCommitments` from it ([#3479](#3479)) ([9c9b561](9c9b561)) * Transcript handled through shared_ptr ([#3434](#3434)) ([30fca33](30fca33)) * Typo fixes ([#3488](#3488)) ([d9a44dc](d9a44dc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.2</summary> ## [0.16.2](AztecProtocol/aztec-packages@aztec-packages-v0.16.1...aztec-packages-v0.16.2) (2023-12-05) ### Features * Add tree snapshots ([#3468](AztecProtocol/aztec-packages#3468)) ([7a86bb3](AztecProtocol/aztec-packages@7a86bb3)) * **AVM:** First version for mini AVM (ADD, RETURN, CALLDATACOPY) ([#3439](AztecProtocol/aztec-packages#3439)) ([b3af146](AztecProtocol/aztec-packages@b3af146)) * Circuit optimized indexed tree batch insertion ([#3367](AztecProtocol/aztec-packages#3367)) ([187d2f7](AztecProtocol/aztec-packages@187d2f7)) * Devnet ([#3473](AztecProtocol/aztec-packages#3473)) ([97c40c2](AztecProtocol/aztec-packages@97c40c2)) * **docs:** Add simple private voting tutorial ([#3402](AztecProtocol/aztec-packages#3402)) ([a6e0352](AztecProtocol/aztec-packages@a6e0352)) * **docs:** Document slow update tree ([#3416](AztecProtocol/aztec-packages#3416)) ([8e9f103](AztecProtocol/aztec-packages@8e9f103)) * Flavor refactor, reduce duplication ([#3407](AztecProtocol/aztec-packages#3407)) ([8d6b013](AztecProtocol/aztec-packages@8d6b013)) * Inclusion and non-inclusion proofs experiment ([#3255](AztecProtocol/aztec-packages#3255)) ([b911e65](AztecProtocol/aztec-packages@b911e65)), closes [#2572](AztecProtocol/aztec-packages#2572) [#2584](AztecProtocol/aztec-packages#2584) * New Poseidon2 circuit builder gates ([#3346](AztecProtocol/aztec-packages#3346)) ([91cb369](AztecProtocol/aztec-packages@91cb369)) * New Poseidon2 relations ([#3406](AztecProtocol/aztec-packages#3406)) ([14b9736](AztecProtocol/aztec-packages@14b9736)) * Pull latest noir for brillig optimizations ([#3464](AztecProtocol/aztec-packages#3464)) ([d356bac](AztecProtocol/aztec-packages@d356bac)) * Refactor StandardIndexedTree for abstract leaves and preimages and optimized it ([#3530](AztecProtocol/aztec-packages#3530)) ([63b9cdc](AztecProtocol/aztec-packages@63b9cdc)) * Removing historical roots from circuits ([#3544](AztecProtocol/aztec-packages#3544)) ([9f682cb](AztecProtocol/aztec-packages@9f682cb)) * Seperate pil files for sub machines ([#3454](AztecProtocol/aztec-packages#3454)) ([d09d6f5](AztecProtocol/aztec-packages@d09d6f5)) * Throw compile time error if contract has too many fns ([#3536](AztecProtocol/aztec-packages#3536)) ([ad66ad0](AztecProtocol/aztec-packages@ad66ad0)) * Use tree snapshots in aztec-node/pxe/oracles ([#3504](AztecProtocol/aztec-packages#3504)) ([6e40427](AztecProtocol/aztec-packages@6e40427)) * Yellow paper cross-chain communication ([#3477](AztecProtocol/aztec-packages#3477)) ([d51df8c](AztecProtocol/aztec-packages@d51df8c)) ### Bug Fixes * Check version, chainid and sender for cross-chain l1 to l2 msgs ([#3457](AztecProtocol/aztec-packages#3457)) ([d251703](AztecProtocol/aztec-packages@d251703)) * **ci:** Add DEPLOY_TAG in fork log group ([#3510](AztecProtocol/aztec-packages#3510)) ([f021041](AztecProtocol/aztec-packages@f021041)) * **ci:** Check if l1 contracts img has been deployed ([#3531](AztecProtocol/aztec-packages#3531)) ([ac1f03c](AztecProtocol/aztec-packages@ac1f03c)) * **ci:** Comment out LB listeners (for now) ([#3519](AztecProtocol/aztec-packages#3519)) ([640aabc](AztecProtocol/aztec-packages@640aabc)) * **ci:** Count for bootnode discovery service ([#3517](AztecProtocol/aztec-packages#3517)) ([2a38788](AztecProtocol/aztec-packages@2a38788)) * **ci:** Define REPOSITORY in deploy_l1_contracts ([#3514](AztecProtocol/aztec-packages#3514)) ([b246d1b](AztecProtocol/aztec-packages@b246d1b)) * **ci:** Don't deploy to npm on master merge ([#3502](AztecProtocol/aztec-packages#3502)) ([a138860](AztecProtocol/aztec-packages@a138860)) * **ci:** Env vars for deploying l1-contracts ([#3513](AztecProtocol/aztec-packages#3513)) ([27106b2](AztecProtocol/aztec-packages@27106b2)) * **ci:** Export FORK_API_KEY from setup_env ([#3512](AztecProtocol/aztec-packages#3512)) ([7e81e2c](AztecProtocol/aztec-packages@7e81e2c)) * **ci:** Fix docker architecture for devnet packages ([#3505](AztecProtocol/aztec-packages#3505)) ([66d0287](AztecProtocol/aztec-packages@66d0287)) * **ci:** Fix faucet vars + don't deploy contracts from node ([#3553](AztecProtocol/aztec-packages#3553)) ([c7176f6](AztecProtocol/aztec-packages@c7176f6)) * **ci:** L1 contracts directories ([#3545](AztecProtocol/aztec-packages#3545)) ([63dd0c8](AztecProtocol/aztec-packages@63dd0c8)) * **ci:** Login to ecr to fetch contracts image ([#3538](AztecProtocol/aztec-packages#3538)) ([b033538](AztecProtocol/aztec-packages@b033538)) * **ci:** Remove unused ADDRESS vars & export private key vars ([#3520](AztecProtocol/aztec-packages#3520)) ([d889359](AztecProtocol/aztec-packages@d889359)) * **ci:** Set default value for $TO_TAINT ([#3508](AztecProtocol/aztec-packages#3508)) ([8b6688a](AztecProtocol/aztec-packages@8b6688a)) * **ci:** Terraform listener resources ([#3534](AztecProtocol/aztec-packages#3534)) ([c3b9cce](AztecProtocol/aztec-packages@c3b9cce)) * **ci:** Terraform_deploy for devnet ([#3516](AztecProtocol/aztec-packages#3516)) ([ba3803e](AztecProtocol/aztec-packages@ba3803e)) * **ci:** Tf variable references & formatting([#3522](AztecProtocol/aztec-packages#3522)) ([d37cf52](AztecProtocol/aztec-packages@d37cf52)) * Disable e2e-slow-tree ([#3459](AztecProtocol/aztec-packages#3459)) ([5927103](AztecProtocol/aztec-packages@5927103)) * **docs:** Update package name of aztec-cli ([#3474](AztecProtocol/aztec-packages#3474)) ([98d7ba0](AztecProtocol/aztec-packages@98d7ba0)) * Double slash in deployed faucet routes ([#3555](AztecProtocol/aztec-packages#3555)) ([6c704a5](AztecProtocol/aztec-packages@6c704a5)) * Faucet lb_listener priority ([#3554](AztecProtocol/aztec-packages#3554)) ([3f56dd7](AztecProtocol/aztec-packages@3f56dd7)) * Handling low_nullifier.next_value equal to 0 ([#3562](AztecProtocol/aztec-packages#3562)) ([c800502](AztecProtocol/aztec-packages@c800502)), closes [#3550](AztecProtocol/aztec-packages#3550) * Remove x86_64 form l1-contracts img tag ([#3549](AztecProtocol/aztec-packages#3549)) ([6828f1a](AztecProtocol/aztec-packages@6828f1a)) * Throw error if fn sig has whitespaces ([#3509](AztecProtocol/aztec-packages#3509)) ([7671063](AztecProtocol/aztec-packages@7671063)), closes [#3055](AztecProtocol/aztec-packages#3055) ### Miscellaneous * (yellow paper) public-vm section of yellow paper ([#3493](AztecProtocol/aztec-packages#3493)) ([8ff3780](AztecProtocol/aztec-packages@8ff3780)) * Add mermaid diagram support ([#3499](AztecProtocol/aztec-packages#3499)) ([537d552](AztecProtocol/aztec-packages@537d552)) * Add yellow paper build check to CI ([#3490](AztecProtocol/aztec-packages#3490)) ([3ebd2f2](AztecProtocol/aztec-packages@3ebd2f2)) * **avm:** Enable AVM unit tests in CI ([#3463](AztecProtocol/aztec-packages#3463)) ([051dda9](AztecProtocol/aztec-packages@051dda9)), closes [#3461](AztecProtocol/aztec-packages#3461) * **bb:** Pointer_view to reference-based get_all ([#3495](AztecProtocol/aztec-packages#3495)) ([50d7327](AztecProtocol/aztec-packages@50d7327)) * **bb:** Reuse entities from GoblinUltra in GoblinUltraRecursive ([#3521](AztecProtocol/aztec-packages#3521)) ([8259636](AztecProtocol/aztec-packages@8259636)) * Build the acir test vectors as part of CI. ([#3447](AztecProtocol/aztec-packages#3447)) ([1a2d1f8](AztecProtocol/aztec-packages@1a2d1f8)) * Containers reduced to ~100MB total. ~30s installation. ([#3487](AztecProtocol/aztec-packages#3487)) ([b49cef2](AztecProtocol/aztec-packages@b49cef2)) * **docs:** Fix broken Noir stdlib link ([#3496](AztecProtocol/aztec-packages#3496)) ([787d59a](AztecProtocol/aztec-packages@787d59a)) * Field-agnostic and reusable transcript ([#3433](AztecProtocol/aztec-packages#3433)) ([d78775a](AztecProtocol/aztec-packages@d78775a)) * Fix broken link in txs in yellow paper ([#3484](AztecProtocol/aztec-packages#3484)) ([798565d](AztecProtocol/aztec-packages@798565d)) * Fix yellow paper build error ([32881a4](AztecProtocol/aztec-packages@32881a4)) * Fixed typo in build system ([#3501](AztecProtocol/aztec-packages#3501)) ([3a80ac2](AztecProtocol/aztec-packages@3a80ac2)) * Increase functions per contract from 16 to 32 ([#3503](AztecProtocol/aztec-packages#3503)) ([ebdeea3](AztecProtocol/aztec-packages@ebdeea3)) * Naming fixes ([#3476](AztecProtocol/aztec-packages#3476)) ([1db30bf](AztecProtocol/aztec-packages@1db30bf)) * Optimise bb.js package size and sandox/cli dockerfiles to unbloat final containers. ([#3462](AztecProtocol/aztec-packages#3462)) ([cb3db5d](AztecProtocol/aztec-packages@cb3db5d)) * Pin node version in docker base images and bump nvmrc ([#3537](AztecProtocol/aztec-packages#3537)) ([5d3895a](AztecProtocol/aztec-packages@5d3895a)) * Recursive verifier updates ([#3452](AztecProtocol/aztec-packages#3452)) ([dbb4a12](AztecProtocol/aztec-packages@dbb4a12)) * Refactor `WitnessEntities` to be able to derive `WitnessCommitments` from it ([#3479](AztecProtocol/aztec-packages#3479)) ([9c9b561](AztecProtocol/aztec-packages@9c9b561)) * Remove temporary logging ([#3466](AztecProtocol/aztec-packages#3466)) ([8c8387b](AztecProtocol/aztec-packages@8c8387b)) * Transcript handled through shared_ptr ([#3434](AztecProtocol/aztec-packages#3434)) ([30fca33](AztecProtocol/aztec-packages@30fca33)) * Typo fixes ([#3488](AztecProtocol/aztec-packages#3488)) ([d9a44dc](AztecProtocol/aztec-packages@d9a44dc)) * **yellow_paper:** Public<>private messaging ([#3491](AztecProtocol/aztec-packages#3491)) ([6ecc406](AztecProtocol/aztec-packages@6ecc406)) ### Documentation * Add transaction section to yellow paper ([#3418](AztecProtocol/aztec-packages#3418)) ([44bf30b](AztecProtocol/aztec-packages@44bf30b)) * Apply comments from Jan on contracts ([#3539](AztecProtocol/aztec-packages#3539)) ([e351873](AztecProtocol/aztec-packages@e351873)) * Fees update in yellow paper ([#3486](AztecProtocol/aztec-packages#3486)) ([a8b2608](AztecProtocol/aztec-packages@a8b2608)) * First go at generated AVM instruction set doc ([#3469](AztecProtocol/aztec-packages#3469)) ([8cc54a4](AztecProtocol/aztec-packages@8cc54a4)) * Further update to the yellow paper ([#3542](AztecProtocol/aztec-packages#3542)) ([751bb6a](AztecProtocol/aztec-packages@751bb6a)) * Yellow paper updates ([#3478](AztecProtocol/aztec-packages#3478)) ([11f754d](AztecProtocol/aztec-packages@11f754d)) * Yellow paper updates for private message delivery ([#3472](AztecProtocol/aztec-packages#3472)) ([6ba9e18](AztecProtocol/aztec-packages@6ba9e18)) * **yellow-paper:** Sync, enqueued, and static calls ([#3494](AztecProtocol/aztec-packages#3494)) ([00835c6](AztecProtocol/aztec-packages@00835c6)), closes [#3108](AztecProtocol/aztec-packages#3108) * **yellowpaper:** Instruction set updates and fixes ([#3515](AztecProtocol/aztec-packages#3515)) ([bfb61dd](AztecProtocol/aztec-packages@bfb61dd)) </details> <details><summary>barretenberg.js: 0.16.2</summary> ## [0.16.2](AztecProtocol/aztec-packages@barretenberg.js-v0.16.1...barretenberg.js-v0.16.2) (2023-12-05) ### Miscellaneous * Optimise bb.js package size and sandox/cli dockerfiles to unbloat final containers. ([#3462](AztecProtocol/aztec-packages#3462)) ([cb3db5d](AztecProtocol/aztec-packages@cb3db5d)) * Pin node version in docker base images and bump nvmrc ([#3537](AztecProtocol/aztec-packages#3537)) ([5d3895a](AztecProtocol/aztec-packages@5d3895a)) </details> <details><summary>barretenberg: 0.16.2</summary> ## [0.16.2](AztecProtocol/aztec-packages@barretenberg-v0.16.1...barretenberg-v0.16.2) (2023-12-05) ### Features * **AVM:** First version for mini AVM (ADD, RETURN, CALLDATACOPY) ([#3439](AztecProtocol/aztec-packages#3439)) ([b3af146](AztecProtocol/aztec-packages@b3af146)) * Flavor refactor, reduce duplication ([#3407](AztecProtocol/aztec-packages#3407)) ([8d6b013](AztecProtocol/aztec-packages@8d6b013)) * New Poseidon2 circuit builder gates ([#3346](AztecProtocol/aztec-packages#3346)) ([91cb369](AztecProtocol/aztec-packages@91cb369)) * New Poseidon2 relations ([#3406](AztecProtocol/aztec-packages#3406)) ([14b9736](AztecProtocol/aztec-packages@14b9736)) * Pull latest noir for brillig optimizations ([#3464](AztecProtocol/aztec-packages#3464)) ([d356bac](AztecProtocol/aztec-packages@d356bac)) * Seperate pil files for sub machines ([#3454](AztecProtocol/aztec-packages#3454)) ([d09d6f5](AztecProtocol/aztec-packages@d09d6f5)) ### Miscellaneous * **avm:** Enable AVM unit tests in CI ([#3463](AztecProtocol/aztec-packages#3463)) ([051dda9](AztecProtocol/aztec-packages@051dda9)), closes [#3461](AztecProtocol/aztec-packages#3461) * **bb:** Pointer_view to reference-based get_all ([#3495](AztecProtocol/aztec-packages#3495)) ([50d7327](AztecProtocol/aztec-packages@50d7327)) * **bb:** Reuse entities from GoblinUltra in GoblinUltraRecursive ([#3521](AztecProtocol/aztec-packages#3521)) ([8259636](AztecProtocol/aztec-packages@8259636)) * Build the acir test vectors as part of CI. ([#3447](AztecProtocol/aztec-packages#3447)) ([1a2d1f8](AztecProtocol/aztec-packages@1a2d1f8)) * Field-agnostic and reusable transcript ([#3433](AztecProtocol/aztec-packages#3433)) ([d78775a](AztecProtocol/aztec-packages@d78775a)) * Optimise bb.js package size and sandox/cli dockerfiles to unbloat final containers. ([#3462](AztecProtocol/aztec-packages#3462)) ([cb3db5d](AztecProtocol/aztec-packages@cb3db5d)) * Pin node version in docker base images and bump nvmrc ([#3537](AztecProtocol/aztec-packages#3537)) ([5d3895a](AztecProtocol/aztec-packages@5d3895a)) * Recursive verifier updates ([#3452](AztecProtocol/aztec-packages#3452)) ([dbb4a12](AztecProtocol/aztec-packages@dbb4a12)) * Refactor `WitnessEntities` to be able to derive `WitnessCommitments` from it ([#3479](AztecProtocol/aztec-packages#3479)) ([9c9b561](AztecProtocol/aztec-packages@9c9b561)) * Transcript handled through shared_ptr ([#3434](AztecProtocol/aztec-packages#3434)) ([30fca33](AztecProtocol/aztec-packages@30fca33)) * Typo fixes ([#3488](AztecProtocol/aztec-packages#3488)) ([d9a44dc](AztecProtocol/aztec-packages@d9a44dc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

sirasistant added 9 commits December 1, 2023 10:44

wip abstracting leaf and preimage in indexed

880b702

feat: experiment with generic indexed trees

ed471d4

chore: add test dep to devDependencies

38fcf05

chore: remove old LeafData struct

bc4915b

fix: getLatestLeafPreimage can return undefined

47a3aee

feat: remove local leaf cache in indexed tree

064d7a8

feat: optimize findLeafIndex for indexed tree

d458756

docs: added jsdoc

ae6b6a0

refactor: low leaf getter returns option

47bb7f0

sirasistant added 4 commits December 4, 2023 10:33

Merge branch 'master' into arv/public_data_tree_in

6d7fc7d

fix: make sure low leaf in range

7250939

fix: improve performance of getLeafIndex

240e8b5

chore: cleanups

bcc8ae0

sirasistant changed the title ~~Arv/public data tree indexed~~ feat: refactor StandardIndexedTree for abstract leaves and preimages and efficient low leaf lookups Dec 4, 2023

sirasistant changed the title ~~feat: refactor StandardIndexedTree for abstract leaves and preimages and efficient low leaf lookups~~ feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it Dec 4, 2023

style: fix formatting

1b7c9f8

sirasistant requested a review from LeilaWang December 4, 2023 12:58

sirasistant added 2 commits December 4, 2023 14:03

Merge branch 'master' into arv/public_data_tree_indexed

3848116

chore: reset file to master

08791c5

sirasistant marked this pull request as ready for review December 4, 2023 14:12

Merge branch 'master' into arv/public_data_tree_indexed

989ad09

LeilaWang reviewed Dec 4, 2023

View reviewed changes

LeilaWang approved these changes Dec 5, 2023

View reviewed changes

sirasistant added 5 commits December 5, 2023 09:38

fix: addressed some PR comments

edb4548

fix: avoid findLeafIndex in sparse trees

e731f97

refactor: avoid createTree helper

037a479

fix: formatting

a217fe2

Merge branch 'master' into arv/public_data_tree_indexed

2a77b8b

sirasistant merged commit 63b9cdc into master Dec 5, 2023
82 checks passed

sirasistant deleted the arv/public_data_tree_indexed branch December 5, 2023 12:58

AztecBot mentioned this pull request Dec 5, 2023

chore(master): Release 0.16.2 #3453

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it #3530

feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it #3530

sirasistant commented Dec 4, 2023 •

edited

Loading

AztecBot commented Dec 4, 2023 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

LeilaWang Dec 4, 2023

LeilaWang Dec 4, 2023 •

edited

Loading

LeilaWang Dec 4, 2023

LeilaWang Dec 4, 2023

LeilaWang Dec 4, 2023

sirasistant Dec 5, 2023

LeilaWang Dec 4, 2023

sirasistant Dec 5, 2023

LeilaWang Dec 4, 2023

sirasistant Dec 5, 2023

feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it #3530

feat: refactor StandardIndexedTree for abstract leaves and preimages and optimized it #3530

Conversation

sirasistant commented Dec 4, 2023 • edited Loading

Checklist:

AztecBot commented Dec 4, 2023 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

Choose a reason for hiding this comment

LeilaWang Dec 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sirasistant commented Dec 4, 2023 •

edited

Loading

AztecBot commented Dec 4, 2023 •

edited

Loading

LeilaWang Dec 4, 2023 •

edited

Loading